
% United States R\tent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Addrew: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Vuginia 22313-1450 
www.aspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



09/535,441 



03/23/2000 



Chinmoy Panda 



07844-364001 



1295 



21876 7590 09/05/2003 

FISH & RICHARDSON P.C. 
500 ARGUELLO STREET 
SUITE 500 

REDWOOD CITY,CA 94063 



EXAMINER 



SMITH, PETER J 



ART UNIT 



PAPER NUMBER 



2176 

DATE MAILED: 09/05/2003 



7 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 07-01) 



Office Action Summary 



Application No. 

09/535,441 



Examiner 

Peter J Smith 



Ilicant(s) 

PANDA, CHINMOY 



Art Unit 

2176 



- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)S Responsive to communication(s) filed on 23 March 2000 . 
2a)D This action is FINAL. 2b)l3 This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) S Claim(s) 1-28 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

Q)M Claim(s) 1-28 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)13 The drawing(s) filed on 23 March 2000 is/are: a)I3 accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
1 !)□ The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)D All b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 . 
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DETAILED ACTION 



1. This action is responsive to communications: application filed on 03/23/2000. 

2. Claims 1-28 are pending in the case. Claim 1, 14, and 27 are independents claim. 



3. The following is a quotation of 35 U.S.C 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 



4. Claims 1-2, 6, 8-15, 19, and 21-28 are rejected under 35 ILS.C. 103(a) as being 
unpatentable over Ho et al. (hereafter referred to as Ho), US 6,021,412 filed 04/02/1996 in 
view of Sciammarella et al. (hereafter referred to as Sciammarella), US 5,982,369 filed 



Regarding independent claim 1, Ho teaches extracting one or more document keywords 
from the document considered important in describing the document in col. 1 lines 50-55. Ho 
also teaches collecting one or more images associated with the document including information 
describing each image in col 1 line 55 - col. 2 line 5. What Ho does not teach is generating a 
proximity factor for each image collected from the document and each document keyword that 
reflects the degree of correlation between the image and the document keyword or determining 
the importance of each image according to an image metric that combines the proximity factors 
for each document keyword and image pair. 



Claim Rejections - 35 USC §103 



04/21/1997. 
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Sciammarella does teach generating a proximity factor for each image collected from the 
document and each document keyword that reflects the degree of correlation between the image 
and the document keyword in Fig. 2 and the abstract. Sciammarella also teaches determining the 
importance of each image according to an image metric that combines the proximity factors for 
each document keyword and image pair in Fig. 4 and the abstract. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the invention as claimed. It would have 
been obvious and desirable to one of ordinary skill in the art at the time of the invention to 
generate a proximity factor relationship between the images and keywords so that user could 
discern between the different relational importance of each keyword and image pair. This would 
have allowed the user to have an additional means for choosing the best image for a particular 
location on the document. 

Regarding dependent claim 2, Ho teaches presenting images determined to be 
important on a display device in col. 6 lines 18-20. 

Regarding dependent claim 6, Ho teaches identifying image text used to describe each 
image in Fig. 5 item 506 and Fig. 6. 

Regarding dependent claim 8, Ho teaches searching the metadata information 
associated with the image for text describing the image in Fig. 5 item 506 and col. 1 lines 47-50. 

Regarding depend claim 9, Ho does not explicitly teach that the metadata information is 
compatible with hypertext markup language. One of ordinary skill in the art at the time of the 
invention would have known how to make metadata compatible with hypertext markup 
language. It would have been obvious to one of ordinary skill in the art at the time the invention 
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was made to combine the their knowledge with Ho to create the claimed invention. It would 
have been obvious and desirable to use hypertext markup language compatible metadata because 
hypertext markup language is the most common form of storing metadata and it would have been 
advantageous to use it to create an invention compatible with well known standards. 

Regarding dependent claim 10, Ho does not explicitly teach lexically analyzing the 
image text associated with each image and each keyword to determine the degree of correlation 
between an image and a keyword. Sciammarella does teach lexically analyzing the image text 
associated with each image and each keyword to determine the degree of correlation between an 
image and a keyword in col. 2 lines 26-30. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the claimed invention. It would have been 
obvious and desirable to determine the degree of correlation between the keywords and the 
images to best determine what image should be placed in each location. 

Regarding dependent claim 11, Ho does not teach performing a phonetic comparison 
between the image text associated with each image and each document keyword to determine the 
degree of correlation between an image and a document keyword. Sciammarella does teach 
performing a phonetic comparison between the image text associated with each image and each 
document keyword to determine the degree of correlation between an image and a document 
keyword in Fig. 2 and col. 2 lines 2-18. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the claimed invention. It would have been 
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obvious and desirable to determine the degree of correlation between each image and keyword 
pair to determine which image is the best match for a particular keyword. 

Regarding dependent claim 12, Ho teaches identifying the location of the image in the 
document in col. 5 lines 14-17 and measuring the distance between the image in the document 
and a keyword in col 6 lines 57-59 as the program knows the positions of both the graphic and 
the keyword on the page. 

Regarding dependent claim 13, Ho teaches determining the correlation between each 
document keyword and an image according to the distance between the document keyword and 
the image in col. 5 lines 14-17. 

Regarding independent claim 14, Ho teaches extracting one or more document 
keywords from the document considered important in describing the document in col. 1 lines 50- 
55. Ho also teaches collecting one or more images associated with the document including 
information describing each image in col. 1 line 55 - col. 2 line 5. What Ho does not teach is 
generating a proximity factor for each image collected from the document and each document 
keyword that reflects the degree of correlation between the image and the document keyword or 
determining the importance of each image according to an image metric that combines the 
proximity factors for each document keyword and image pair. 

Sciammarella does teach generating a proximity factor for each image collected from the 
document and each document keyword that reflects the degree of correlation between the image 
and the document keyword in Fig. 2 and the abstract. Sciammarella also teaches determining the 
importance of each image according to an image metric that combines the proximity factors for 
each document keyword and image pair in Fig. 4 and the abstract. 
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It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the invention as claimed. It would have 
been obvious and desirable to one of ordinary skill in the art at the time of the invention to 
generate a proximity factor relationship between the images and keywords so that user could 
discern between the different relational importance of each keyword and image pair. This would 
have allowed the user to have an additional means for choosing the best image for a particular 
location on the document. 

Regarding dependent claim 15, Ho teaches presenting images determined to be 
important on a display device in col. 6 lines 18-20. 

Regarding dependent claim 19, Ho teaches identifying image text used to describe each 
image in Fig. 5 item 506 and Fig. 6. 

Regarding dependent claim 21, Ho teaches searching the metadata information 
associated with the image for text describing the image in Fig. 5 item 506 and col. 1 lines 47-50. 

Regarding depend claim 22, Ho does not explicitly teach that the metadata information 
is compatible with hypertext markup language. One of ordinary skill in the art at the time of the 
invention would have known how to make metadata compatible with hypertext markup 
language. It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine the their knowledge with Ho to create the claimed invention. It would 
have been obvious and desirable to use hypertext markup language compatible metadata because 
hypertext markup language is the most common form of storing metadata and it would have been 
advantageous to use it to create an invention compatible with well known standards. 
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Regarding dependent claim 23, Ho does not explicitly teach lexically analyzing the 
image text associated with each image and each keyword to determine the degree of correlation 
between an image and a keyword. Sciammarella does teach lexically analyzing the image text 
associated with each image and each keyword to determine the degree of correlation between an 
image and a keyword in col. 2 lines 26-30. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the claimed invention. It would have been 
obvious and desirable to determine the degree of correlation between the keywords and the 
images to best determine what image should be placed in each location. 

Regarding dependent claim 24, Ho does not teach performing a phonetic comparison 
between the image text associated with each image and each document keyword to determine the 
degree of correlation between an image and a document keyword. Sciammarella does teach 
performing a phonetic comparison between the image text associated with each image and each 
document keyword to determine the degree of correlation between an image and a document 
keyword in Fig. 2 and col. 2 lines 2-18. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the claimed invention. It would have been 
obvious and desirable to determine the degree of correlation between each image and keyword 
pair to determine which image is the best match for a particular keyword. 

Regarding dependent claim 25, Ho teaches identifying the location of the image in the 
document in col. 5 lines 14-17 and measuring the distance between the image in the document 
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and a keyword in col. 6 lines 57-59 as the program knows the positions of both the graphic and 
the keyword on the page. 

Regarding dependent claim 26, Ho teaches determining the correlation between each 
document keyword and an image according to the distance between the document keyword and 
the image in col. 5 lines 14-17. 

Regarding independent claim 27, Ho teaches extracting one or more document 
keywords from the document considered important in describing the document in col. 1 lines 50- 
55. Ho also teaches collecting one or more images associated with the document including 
information describing each image in col. 1 line 55 - col. 2 line 5. What Ho does not teach is 
generating a proximity factor for each image collected from the document and each document 
keyword that reflects the degree of correlation between the image and the document keyword or 
determining the importance of each image according to an image metric that combines the 
proximity factors for each document keyword and image pair. 

Sciammarella does teach generating a proximity factor for each image collected from the 
document and each document keyword that reflects the degree of correlation between the image 
and the document keyword in Fig. 2 and the abstract. Sciammarella also teaches determining the 
importance of each image according to an image metric that combines the proximity factors for 
each document keyword and image pair in Fig. 4 and the abstract. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the invention as claimed. It would have 
been obvious and desirable to one of ordinary skill in the art at the time of the invention to 
generate a proximity factor relationship between the images and keywords so that user could 
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discern between the different relational importance of each keyword and image pair. This would 
have allowed the user to have an additional means for choosing the best image for a particular 
location on the document. 

Regarding dependent claim 28, Ho teaches presenting images determined to be 
important on a display device in col. 6 lines 18-20. 

5. Claims 3-5 and 16-18 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ho et al. (hereafter referred to as Ho), US 6,021,412 filed 04/02/1996 in view of 
Sciammarella et aL (hereafter referred to as Sciammarella), US 5,982,369 filed 04/21/1997 
as applied to claims 1 and 14 above, and further in view of Dutta, US 6,480,837 Bl filed 
12/16/1999. 

Regarding dependent claim 3, Ho does not teach ordering the document keywords 
according to an ordering criterion or weighting the proximity factor associated with each 
document keyword and image pair based on the order of the document keyword. Dutta does 
teach ordering the document keywords according to an ordering criterion and weighting the 
proximity factor associated with each document keyword and image pair based on the order of 
the document keyword in Fig. 2. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Sciammarella to create the claimed invention. It 
would have been obvious and desirable to order and weight the keywords according to a 
popularity weight so that some of the keywords would have had an increased chance of gaining 
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an image located near them. This would have been desirable to give the user an ability to 
accentuate certain parts of the document over others with the use of images. 

Regarding dependent claim 4, Ho does not teach that the frequency that each document 
keyword appears in the document determines the ordering criterion used to order the document 
keywords. Dutta does teach that the frequency that each document keyword appears in the 
document determines the ordering criterion used to order the document keywords in col. 1 lines 
23-37. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Sciammarella to create the invention as claimed. 
It would have been obvious and desirable to use the frequency of the keywords to determine the 
location the image should be placed on the document. It is common sense to place the image 
where the keyword appears most frequently. 

Regarding dependent claim 5, Ho does not teach that the ordering criterion orders the 
document keywords according to their relationship with the subject matter of the document. 
Dutta does teach that the ordering criterion orders the document keywords according to their 
relationship with the subject matter of the document in col. 4 lines 25-42. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Sciammarella to create the claimed invention. It 
would have been obvious and desirable to order the keywords according to the relationship with 
the subject matter so that the images are matched with the keywords with which they have the 
strongest relationship. 



Application/Control Number: 09/535,441 Page 1 1 

Art Unit: 2176 

Regarding dependent claim 16, Ho does not teach ordering the document keywords 
according to an ordering criterion or weighting the proximity factor associated with each 
document keyword and image pair based on the order of the document keyword. Dutta does 
teach ordering the document keywords according to an ordering criterion and weighting the 
proximity factor associated with each document keyword and image pair based on the order of 
the document keyword in Fig. 2. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Sciammarella to create the claimed invention. It 
would have been obvious and desirable to order and weight the keywords according to a 
popularity weight so that some of the keywords would have had an increased chance of gaining 
an image located near them. This would have been desirable to give the user an ability to 
accentuate certain parts of the document over others with the use of images. 

Regarding dependent claim 17, Ho does not teach that the frequency that each 
document keyword appears in the document determines the ordering criterion used to order the 
document keywords. Dutta does teach that the frequency that each document keyword appears 
in the document determines the ordering criterion used to order the document keywords in col. 1 
lines 23-37. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Sciammarella to create the invention as claimed. 
It would have been obvious and desirable to use the frequency of the keywords to determine the 
location the image should be placed on the document. It is common sense to place the image 
where the keyword appears most frequently. 
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Regarding dependent claim 18, Ho does not teach that the ordering criterion orders the 
document keywords according to their relationship with the subject matter of the document. 
Dutta does teach that the ordering criterion orders the document keywords according to their 
relationship with the subject matter of the document in col. 4 lines 25-42. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Sciammarella to create the claimed invention. It 
would have been obvious and desirable to order the keywords according to the relationship with 
the subject matter so that the images are matched with the keywords with which they have the 
strongest relationship. 

6. Claims 7 and 20 are rejected under 35 U.S.C 103(a) as being unpatentable over Ho 
et al. (hereafter referred to as Ho), US 6,021,412 filed 04/02/1996 in view of Sciammarella et 
al. (hereafter referred to as Sciammarella), US 5,982,369 filed 04/21/1997 as applied to 
claims 6 and 19 above, and further in view of Matsumoto, US 6,526,170 filed 12/13/1994. 

Regarding dependent claim 7, Ho does not teach scanning a bit-mapped representation 
of the image for text information or converting the bit-mapped representation of the text 
information into image text. Matsumoto does teach scanning a bit-mapped representation of the 
image for text information and converting the bit-mapped representation of the text information 
into image text in Fig. 7 and col. 1 lines 23-25. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Matsumoto into Ho in view of Sciammarella to create the claimed 
invention. It would have been obvious and desirable to extract text from the image itself and use 
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that to match with the keywords so that the image-keyword match would further increase in 
accuracy. 

Regarding dependent claim 20, Ho does not teach scanning a bit-mapped representation 
of the image for text information or converting the bit-mapped representation of the text 
information into image text. Matsumoto does teach scanning a bit-mapped representation of the 
image for text information and converting the bit-mapped representation of the text information 
into image text in Fig. 7 and col. 1 lines 23-25. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Matsumoto into Ho in view of Sciammarella to create the claimed 
invention. It would have been obvious and desirable to extract text from the image itself and use 
that to match with the keywords so that the image-keyword match would further increase in 
accuracy. 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Peter J Smith whose telephone number is 703-305-5931. The 
examiner can normally be reached on Mondays-Fridays 7:00am-3:30pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph H Feild can be reached on 703-305-9792. The fax phone numbers for the 
organization where this application or proceeding is assigned are 703-746-7239 for regular 
communications and 703-746-7238 for After Final communications. 



Conclusion 
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Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-305-3900. 



PJS 

August 8, 2003 




'JOSEPH H. FEILD 
PRIMARY EXAMINER 



